User experience evaluation

User experience (UX) evaluation means investigating how a person feels about using a system (product, service, non-commercial item, or a combination of them). It is non-trivial to evaluate user experience and come up with solid results, since user experience is subjective, context-dependent and dynamic over time[1]. Laboratory experiments may work well for studying a specific aspect of user experience, but holistic user experience is optimally studied over a longer period of time with real users in a natural environment.

Detailed guidance for user experience evaluation is hard to give, since there are many different dimensions to consider when choosing the evaluation approach:

In all cases, however, there are certain aspects of user experience that evaluators are interested in (measures), and certain procedures and techniques used for collecting the data (methods). When investigating user experience evaluation methods, we can identify methods for emotion assessment and overall UX assessment. The measures and methods for these two evaluation types are described below. Episodic UX can be evaluated with either approach, depending on the case.

Contents

Emotion evaluation

When investigating momentary user experiences, we can evaluate the level of positive affect, negative affect, joy, surprise, frustration, etc. The measures for emotions are bound to the methods used for emotion assessment, but typical emotion measures are e.g. valence and arousal. Objective emotion data can be collected by psychophysiological measurements or by observing expressed emotions. Subjective emotional data can be collected by using self-report methods, which can be verbal or non-verbal.

Examples of emotion evaluation methods:

Overall UX evaluation

In contrast to identifying a momentary emotion, overall UX evaluation investigates how a person feels about a system as a whole, typically after using it for a while. Many of the overall UX evaluation methods are suitable also for evaluating episodic UX, i.e., assessing how a person feels about a specific interaction episode or after executing a task.

There is no agreement on the exact measures for evaluating the overall UX with a system, largely because different products aim at different kinds of experiences. However, there are some high-level constructs of user experience that can be used as the basis for defining the user experience measures, for example:

  1. Utility: Does the user perceive the functions in the system as useful and fit for the purpose?
  2. Usability: Does the user feel that it is easy and efficient to get things done with the system?
  3. Aesthetics:[6] Does the user see the system as visually attractive? Does it feel pleasurable in hand?
  4. Identification: Can I identify myself with the product? Do I look good when using it?
  5. Stimulation: Does the system give me inspiration? Or wow experiences?
  6. Value: Is the system important to me? What is its value for me?

Since the importance of the above user experience constructs is different to different people, it is an interesting option to define the overall UX measures together with each study participant. Another option to evaluate overall UX is to use simply a scale from positive to negative, without further consideration of the user experience constructs.

Overall UX assessment is methodologically different from objective emotion assessment, but similar to subjective emotion assessment. Generic subjective user experience evaluation methods include interviews, questionnaires, story-telling, and often, a combination of these. An individual method can collect data about a set of specific constructs of user experience, such as usability testing is to collect data about usability construct.

Examples of overall UX evaluation methods (excluding traditional usability methods):

UX in video games

A relatively new pursuit in video game playtesting is UX and Usability research. An increasing amount of companies including some of the world's biggest publishers have begun outsourcing UX evaluation or opening their own in-house labs[11] [12][13]. Researchers use a variety of HCI and psychological techniques to examine the effectiveness of the User Experience of the games during the design process[14]. There are also some companies starting to use biometrics to scientifically measure the relationship between in-game events and the player's emotions and feelings (the UX), such as Vertical Slice and Serco ExperienceLab in the UK[15][16], and Valve Software, Electronic Arts, BoltPeters, and VMC Labs in the USA and Canada[17][18][19][20]. The interest in this area comes from both academia and industry, sometimes enabling collaborative work[21][22]. Game UX work has been featured at professional venues, such as the Game Developers Conference (GDC)[23][24]

See also

References

  1. ^ Law, E., Roto, V., Hassenzahl, M., Vermeeren, A., Kort, J.: Understanding, Scoping and Defining User Experience: A Survey Approach. In Proceedings of Human Factors in Computing Systems conference, CHI’09. 4–9 April 2009, Boston, MA, USA (2009)
  2. ^ Baenziger, T., Tran, V. and Scherer,K.R. (2005) ‘‘The EmotionWheel. A Tool for the Verbal Report of Emotional Reactions’’, poster presented at the conference of the International Society of Research on Emotion, Bari, Italy.
  3. ^ Laurans, G., Desmet, P.M.A., & Hekkert, P.P.M. (2009). The emotion slider: a self-report device for the continuous measurement of emotion. Proceedings of the 2009 International Conference on Affective Computing and Intelligent Interaction. Amsterdam, The Netherlands.
  4. ^ Isbister, K., Höök, K., Sharp, M., and Laaksolahti, J. 2006. The sensual evaluation instrument: developing an affective evaluation tool. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada, 22–27 April 2006). CHI '06. ACM, New York, NY, 1163–1172
  5. ^ Desmet, P.M.A., Overbeeke, C.J., Tax, S.J.E.T. (2001). Designing products with added emotional value: development and application of an approach for research through design. The Design Journal, 4(1), 32–47.
  6. ^ Moshagen, M. & Thielsch, M. T. (2010). Facets of visual aesthetics. In: International Journal of Human-Computer Studies, 68 (10), 689–709. PDF
  7. ^ Bolger, N., Davis, A., & Rafaeli, E. (2003). Diary methods: Capturing life as it is lived. Annual Review of Psychology, 54, 579–616.
  8. ^ Csikszentmihalyi M, Larson R. (1987). Validity and reliability of the Experience-Sampling Method. Journal of Nervous and Mental Diseases. Sep 1987;175(9):526–536.
  9. ^ Kahneman, D., Krueger, A., Schkade, D., Schwarz, N., and Stone, A. (2004). A Survey Method for Characterizing Daily Life Experience: The Day Reconstruction Method. Science. 306:5702, pp. 1776–780.
  10. ^ Hassenzahl, M., Burmester, M., & Koller, F. (2003). AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In J.Ziegler & G. Szwillus (Eds.), Mensch & Computer 2003. Interaktion in Bewegung (pp. 187–196). Stuttgart, Leipzig: B.G. Teubner.
  11. ^ Halo 3: How Microsoft Labs Invented a New Science of Play. Wired.com. Retrieved on 21 October 2011.
  12. ^ Bolt, Nate. (2009-01-22) Researching Video Games the UX Way – Boxes and Arrows: The design behind the design. Boxes and Arrows. Retrieved on 21 October 2011.
  13. ^ THQ Chooses The Guildhall at SMU to House New Usability Lab | games industry | MCV. Mcvuk.com. Retrieved on 21 October 2011.
  14. ^ Hong, T. (2008) Shoot to Thrill: Bio-Sensory Reactions to 3D Shooting Games, Game Developer Magazine, October
  15. ^ Video Game Usability and User Experience. Vertical Slice. Retrieved on 21 October 2011.
  16. ^ Game usability testing. PlayableGames. Retrieved on 21 October 2011.
  17. ^ Valve. Valvesoftware.com. Retrieved on 21 October 2011.
  18. ^ EA Games – Electronic Arts. Ea.com. Retrieved on 21 October 2011.
  19. ^ VMC Consulting – Tailored Solutions for Your Business. Vmc.com. Retrieved on 21 October 2011.
  20. ^ Bolt | Peters | Research, design, and products. Boltpeters.com. Retrieved on 21 October 2011.
  21. ^ Nacke, L., Ambinder, M., Canossa, A., Mandryk, R., Stach, T. (2009). "Game Metrics and Biometrics: The Future of Player Experience Research“ Panel at Future Play 2009
  22. ^ 8–9 April 2010, Seminar Presentation at Games Research Methods Seminar, "Using physiological measures in conjunction with other UX approaches for better understanding of the player’s gameplay experiences", University of Tampere, Finland
  23. ^ Ambinder, M. (2011) Biofeedback in Gameplay: How Valve Measures Physiology to Enhance Gaming Experience. Game Developers Conference 2011
  24. ^ Zammitto, V. (2011) The Science of Play Testing: EA's Methods for User Research. Game Developers Conference 2011